1. Field of the Invention
The invention relates to neural networks used for pattern recognition, and more particularly to Hopfield and BAM networks having improved memory capacity.
2. Description of the Related Art
Conventional digital computer systems have become extremely capable. They have large memory and storage capacities and very high speeds. However, there are many areas where conventional computing techniques do not provide satisfactory solutions. Even the great speeds of modern parallel processing systems are insufficient. One of these areas is pattern recognition.
A new class of computing devices, called neural networks, has developed. One area where neural networks have shown a major advantage over conventional computing techniques is pattern recognition. The devices are called neural networks because their operation is based on the operation and organization of neurons. In general, the output of one neuron is connected to the input of many other neurons, with a weighting factor being applied to each input. The weighted inputs are then summed and commonly provided to threshold comparison logic, to indicate on or off. An output is then provided and this may continue to the next level or may be the final output.
Neural networks can be implemented in specific circuits, a hardware implementation, or may be implemented in a computer program or software techniques. Numerous neural networks have been developed. The most common network involves an input layer, a hidden layer and an output layer, with various connections from layer to layer and feedback if desired. Each neuron in each layer performs the input weighting, summing and thresholding functions. Another network class is the bi-directional associative memory (BAM), which includes a further variation, the Hopfield network. A BAM is a two layer network, with the neurons of one layer receiving all the outputs of the other layer, but none from its own layer. In a Hopfield network, there is only one layer of neurons, each receiving the outputs from all the neurons, including itself.
As noted above, the inputs are provided to a weighting system. One difficulty in the use of neural networks is the development of the weights. This typically requires a learning technique and certain learning rules. By far the most common and fundamental learning rule is Hebb's Rule: EQU .DELTA.W.sub.ij =A.sub.i O.sub.j
where .DELTA.W.sub.ij is the weight change for the neuron j to neuron i link, A.sub.i is the activation value for neuron i and O.sub.j is the output of neuron j.
One major problem with Hebbian learning is that it typically results in a very small storage capacity, given the number of neurons. Thus Hebbian learning is very inefficient of neurons. Other improved learning techniques for BAMs and Hopfield networks still have the problem of small storage capacity, in view of the number of neurons.
With this very low storage density, use of BAM and Hopfield networks has been limited. While the primary use of such networks is pattern and character recognition, the very limited capacity of BAM and Hopfield networks has limited their use. If the memory capacity were increased, without overly sacrificing differentiation, then application of these two networks could greatly increase.